Activation functions in deep learning: A comprehensive survey and benchmark

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural been introduced deal with different However, the main goal any network is transform non-linearly separable input data into more linearly abstract features using a hierarchy layers. These layers are combinations linear and nonlinear functions. The most popular common non-linearity activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish Mish. In this paper, comprehensive overview survey presented for AFs deep learning. Different classes Sigmoid Tanh based, ReLU ELU Learning based covered. Several characteristics output range, monotonicity, smoothness also pointed out. A performance comparison performed among 18 state-of-the-art on data. insights benefit researchers doing further research practitioners select choices. code used experimental released at: https://github.com/shivram1987/ActivationFunctions.